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Description 

DUAL DISIMILAR SENSING OBJECT 
DETECTION AND TARGETING SYSTEM 

Background of Invention 

[0001] The present invention relates to collision warning and 
countermeasure systems for an automotive vehicle, and 
more particularly to a system and method of performing 
object detection and association using multiple similar 
and dissimilar sensors. 

[0002] Collision warning and countermeasure systems are be- 
coming more widely used. Collision warning systems 
preferably are able to detect an object approximately 
50m-100m from a host vehicle and assess whether the 
object detected is an obstacle and poses a threat to the 
host vehicle. These systems also preferably provide a ve- 
hicle operator knowledge and awareness of obstacles or 
vehicles within a close proximity in time such that the op- 
erator may perform actions to prevent colliding with the 
detected obstacles. Countermeasure systems exist in vari- 



ous passive and active forms. Some countermeasure sys- 
tems are used to aid in prevention of a collision others are 
used to aid in the prevention of injury to a vehicle opera- 
tor. 

[0003] Certain collision warning systems and countermeasure 
systems are able to sense a vehicle or object in a close 
proximity to a host vehicle and warn a host vehicle opera- 
tor, such that the operator can take precautionary steps to 
prevent a collision or injury. The collision warning systems 
maybe forward or rearward sensing. These systems indi- 
cate to a vehicle operator that an object, that may not be 
visible to the vehicle operator, is within a stated distance 
and location relative to the host vehicle. The vehicle oper- 
ator may than respond accordingly. Other collision warn- 
ing systems and countermeasure systems activate passive 
countermeasures such as air bags, load limiting seat 
belts, or active vehicle control including steering control, 
accelerator control, or brake control whereby the system 
itself aids in preventing a collision or injury. 

[0004] Separate from collision warning systems and countermea- 
sure systems other systems exist for autonomously con- 
trolling a vehicle. These systems may include sensors, 
such as cameras, radar sensors, or ultrasonic sensors to 



gather information and utilize this information when acti- 
vating steering control, brake control, and accelerator 
control. Autonomous type systems have primarily been 
used for military applications but have not been used for 
either providing collision warning or injury prevention in 
automotive vehicles. 
[0005] | n threat estimation, detection, and identification of an 

object in a host vehicle's path, multiple sensors of various 
types have been used. The sensors have been used to de- 
tect an object followed by use of parametric association/ 
estimation techniques to confirm when an object is a po- 
tential threat to the host vehicle and to host vehicle occu- 
pants. The object of concern may be a real object or a 
false object. False objects may be detected for example, 
when there is a stationary roadside object that is foreseen 
as a true potentially collision causing object or when a 
small object, that is not a potential threat, is in the path of 
the host vehicle and is identified and misclassified as a 
potentially collision-causing object. Another example of 
false object generation is when a ghost object is falsely 
generated by the collision warning and countermeasure 
system that in actuality does not exist, as known in the 
art. 



[0006] The collision warning and countermeasure systems collect 
data from multiple sensors and associate, fuse, or com- 
bine the data to determine whether detected objects are 
real objects rather than false objects. Advantages of uti- 
lizing data from multiple sensors includes extended spa- 
tial and temporal coverage, increased accuracy in deter- 
mining whether an object is a potential threat, and in- 
creased reliability in detection of objects in a close prox- 
imity of the host vehicle, thereby, providing a better as- 
sessment of the vehicle's surroundings. 

[0007] choice of architecture for processing object information, 
being cost effective, and being able to provide relevant 
information quick enough so that appropriate action can 
be performed is critical in determining whether a particu- 
lar technology is feasible for use. There is a current desire 
to use cameras in detecting and classifying objects. Un- 
fortunately, current camera technology is not cost effec- 
tive and requires a large amount of processing power and 
time to compute relevant information required for in- 
vehicle use. For example, image processing from a charge 
coupled device (CCD) camera is time consuming due to a 
large amount of data collected for each image, approxi- 
mately 640x480 pixels per frame at 30 frames per sec- 



ond. Time is critical in prevention of a collision. To accu- 
rately classify and track an object can require acquisition 
of tens to hundreds of frames of data each of which hav- 
ing a minimum desired resolution, thus, rendering current 
camera technology for in-vehicle use infeasible. 
[0008] a desire exists to provide a safer automotive vehicle with 
increased collision warning and safety countermeasure in- 
telligence as to decrease the probability of a collision or of 
an injury. It is therefore desirable to provide an improved 
collision warning and safety countermeasure system for 
an automotive vehicle that incorporates use of multiple 
sensor technologies and provides a more efficient colli- 
sion and injury prevention system for an automotive vehi- 
cle. 

Summary of Invention 

[0009] The present invention provides a system and method of 
detecting objects within close proximity of an automotive 
vehicle using both object detection and image generation 
technologies such as electro-magnetics and electro-opti- 
cal technologies, respectively. A method of performing 
object detection within a collision warning and counter- 
measure system is provided. The method includes captur- 
ing & generating an object detection signal and generat- 



ing an image detection signal. Centers-of-reflection and 
centers-of-intensity are determined in response to the 
object detection signal and the image detection signal. 
The centers-of-intensity are associated with the centers- 
of-reflection for a plurality of frames. Differences between 
the centers-of-reflection and the centers-of-intensity are 
determined for each frame and a sensor difference signal 
is generated. An object is classified in response to the 
sensor difference signal. 

[0010] one of several advantages of the present invention is that 
it provides a collision warning and countermeasure sys- 
tem that minimizes image processing time, processor re- 
quirements, memory requirements, & complexity, thus 
providing a cost effective and feasible solution for in- 
vehicle use of existing camera technology. 

[0011] Another advantage of the present invention is that it pro- 
vides a method of associating data collected from both 
electro-magnetic and electro-optical sensors, such as 
radar sensors and cameras, as to better and more effi- 
ciently classify and track objects. 

[0012] Furthermore the present invention continuously adjusts a 
region-of-interest and corresponding center-of-intensity 
for each image frame to provide better certainty of an ob- 



ject and improved tracking capability. 
[0013] The present invention itself, together with attendant ad- 
vantages, will be best understood by reference to the fol- 
lowing detailed description, taken in conjunction with the 

accompanying figures. 
Brief Description of Drawings 

[0014] For a more complete understanding of this invention ref- 
erence should now be had to the embodiments illustrated 
in greater detail in the accompanying figures and de- 
scribed below by way of examples of the invention 
wherein: 

[0015] Figure 1 is a block diagrammatic view of a collision warn- 
ing and countermeasure system for an automotive vehicle 
in accordance with an embodiment of the present inven- 
tion. 

[0016] Figure 2A is a logic flow diagram illustrating a method of 
performing object detection within the collision warning 
and countermeasure system in accordance with an em- 
bodiment of the present invention. 

[0017] Figure 2B is a continuation of the logic flow diagram of 
Figure 2A in accordance with an embodiment of the 
present invention. 

[0018] Figure 3 is a sample of an electromagnetic sensor sweep 



plot illustrating range and azimuth angles of detected ob- 
jects in accordance with an embodiment of the present in- 
vention. 

[0019] Figure 4 is a sample image frame corresponding to the 
detected objects of Figure 3 in accordance with an em- 
bodiment of the present invention. 

[0020] Figure 5 is a sample of an image frame with perspective 
transform, illustrating sample regions-of-interest, cen- 
ters-of-reflection, and centers-of-intensity in accordance 
with an embodiment of the present invention. 

[0021] Figure 6 is an image frame diagram illustrating sample 
radar frames and associated image frames in accordance 
with an embodiment of the present invention. 

[0022] Figure 7 A is a diagram of a sample image frame illustrat- 
ing updated regions-of-interest for objects becoming 
closer to a host vehicle in accordance with an embodiment 
of the present invention; and. 

[0023] Figure 7B is a sample image frame illustrating predicted 

regions-of-interest for objects moving away from the 

host vehicle in accordance with an embodiment of the 

present invention. 
Detailed Description 

[0024] | n th e following figures the same reference numerals will 



be used to refer to the same components. While the 
present invention is described with respect to system and 
method of detecting and tracking objects within close 
proximity of an automotive vehicle using both electro- 
magnetic detection and electro-optical technologies, the 
present invention may be adapted and applied to various 
systems including: collision warning systems, collision 
avoidance systems, parking aid systems, reversing aid 
systems, countermeasure systems, vehicle systems, or 
other systems that may require collision avoidance or as- 
sessment. 

[0025] | n the following description, various operating parameters 
and components are described for one constructed em- 
bodiment. These specific parameters and components are 
included as examples and are not meant to be limiting. 

[0026] Also, in the following description the term "performing" 
may include activating, deploying, initiating, powering, 
and other terms known in the art that may describe the 
manner in which a passive countermeasure may be oper- 
ated. 

[0027] Additionally, in the following description the term "coun- 
termeasure" may refer to reversible or irreversible coun- 
termeasures. Reversible countermeasures refer to coun- 



termeasures that may be reset to their original form or 
used repeatedly without a significant amount of functional 
deficiency, which may be determined by a system de- 
signer. Irreversible countermeasures refer to countermea- 
sures such as airbags that, once deployed, are not 
reusable. 

[0028] Moreover, a countermeasure signal may include informa- 
tion pertaining to the above stated reversible and irre- 
versible countermeasures or may include other informa- 
tion, such as collision warning information. For example, 
the countermeasure signal may contain object detection 
information, which may be used to indicate to a vehicle 
operator the presence or close proximity of a detected 
object. 

[0029] Referring now to Figure 1, a block diagrammatic view of a 
collision warning and countermeasure system 10 for an 
automotive vehicle or host vehicle 12 in accordance with 
an embodiment of the present invention is shown. The 
system 10 includes electro-magnetic sensors 14 and at 
least one electro-optical sensor 16, which are electrically 
coupled to a main controller 18. The controller 18 com- 
bines information received from the electro-magnetic 
sensors 14 and the electro-optical sensor 16 to detect 



and associate objects from multiple sensors for the pur- 
pose of object tracking and threat assessment within a 
close proximity of the vehicle 12. The system 10 may also 
include various vehicle dynamic sensors 20, active coun- 
termeasures 22, passive countermeasures 24, and an in- 
dicator 26, which are all electrically coupled to the con- 
troller 18. The main controller 18 may activate the coun- 
termeasures 22 and 24 or indicate to a vehicle operator 
various object and vehicle information, via the indicator 
26, to prevent a vehicle collision and injury to vehicle oc- 
cupants. 

[0030] | n determining sensors that are appropriate for a given 
application, factors such as range, range rate, shape, and 
size of an object are considered. In a preferred embodi- 
ment of the present invention active sensors in the form 
of radar are used for electro-magnetic sensors 14 and a 
passive sensor in the form of a camera is used for the 
electro-optical sensor 16 to access host vehicle 12 sur- 
roundings. Radar provides derived measurements such as 
range, range rate, azimuth angle, elevation, and approxi- 
mate size of an object, as well as other information known 
in the art. A camera may provide derived measurements 
such as location, size, and shape of an object. 



[0031] The electro-magnetic sensors 14 may be of various sen- 
sor technologies including radar, lidar, or other sensor 
technology forms known in the art and may be referred to 
as active sensors. The electro-magnetic sensors 14 gen- 
erate multiple object detection signals (RCS, Frequency & 
time) upon detecting one or more objects of various size 
and shape. Although four electro-magnetic sensors are 
shown, any number of electro-magnetic sensors may be 
utilized. In the present invention, the object detection sig- 
nals are utilized to compute derived measurements such 
as object's relative range, azimuth angle, velocity, and 
bearing information, as well as other object information 
known in the art. 

[0032] The electro-optical sensor 16, although, is preferably a 

charge-coupled device (CCD) camera it may be of another 
type such as a camera using complementary metal-oxide 
semiconductor (CMOS) technology. The electro-optical 
sensor 16 may be referred to as a passive sensor. The 
electro-optical sensor 16 is a two-dimensional device that 
may have varying resolution, accuracy, field-of-view 
(FOV), and silicon wafer capability. 

[0033] The electro-magnetic sensors 14 may be considered as 
primary sensors and the electro-optical sensors 16 may 



be considered as secondary sensors. 
[0034] The main controller 18 is preferably microprocessor based 
such as a computer having a central processing unit, 
memory (RAM and/or ROM), and associated input and 
output buses. The main controller 18 may be a portion of 
a central vehicle main control unit, an interactive vehicle 
dynamics module, a restraints control module, a main 
safety controller, or may be a stand-alone controller as 
shown. 

[0035] The vehicle dynamics sensors 20 may include a transmis- 
sion rotation sensor, awheel speed sensor, an accelerom- 
eter, an optical sensor, or other velocity or acceleration 
sensors known in the art. The vehicle dynamic sensors 20 
are used to determine velocity and acceleration of the ve- 
hicle 12 and generate a vehicle dynamics signal. 

[0036] Active countermeasures 22 may include control of a brake 
system 22a, a drivetrain system 22b, a steering system 
22c, a chassis system 22d control, and other active coun- 
termeasures known in the art. 

[0037] The passive countermeasures 24 may include passive 
countermeasures, such as air bags 24a, pretensioners 
24b, inflatable seat belts 24c, a load limiting pedal and 
steering column 24d, and other passive countermeasures 



and control thereof as known in the art. Some possible 
passive countermeasures that may be included, but that 
are not shown are seatbelt control, knee bolster control, 
head restraint control, load limiting pedal control, load 
limiting steering control, pretensioner control, external 
airbag control, and pedestrian protection control. Preten- 
sioner control may include control over pyrotechnic and 
motorized seatbelt pretensioners. Airbag control may in- 
clude control over front, side, curtain, hood, dash, or 
other type airbags. Pedestrian protection control may in- 
clude controlling a deployable vehicle hood, a bumper 
system, or other pedestrian protective devices. 

[0038] indicator 26 is used to signal or indicate a collision-warn- 
ing signal or an object identification signal in response to 
the object detection signals. The indicator 26 may include 
a video system, an audio system, an LED, a light, global 
positioning system, a heads-up display, a headlight, a 
taillight, a display system, a telematic system or other in- 
dicator. The indicator 26 may supply warning signals, col- 
lision-related information, external-warning signals to 
objects or pedestrians located outside of the vehicle 12, 
or other pre and post collision information. 

[0039] Referring now to Figures 2A-B, a logic flow diagram illus- 



trating a method of performing object detection within the 
collision warning and countermeasure system 10 is 
shown. 

[0040] | n ste p iqo, the system 10 is initialized. The controller 18 
determines whether radar data is available at a frequency 
of approximately 2Hz. When radar data is available the 
controller 18 proceeds to step 101, otherwise the con- 
troller 18 recognizes that a fault may exist and indicates 
an error. Also, radar frame designator j is set equal to 
one. 

[0041] | n s tep 101, the electro-magnetic sensors 14 may gener- 
ate one or more object detection signals. The electro- 
magnetic sensors 14 perform upramp sweeps 30 and a 
downramp sweeps 32. The object detection signals con- 
tain Object designators 34 that are generated in both the 
upramp sweeps 30 and downramp sweeps 32 and corre- 
spond to detected objects 36 obtained by comparing a re- 
ceived echo signal with an originally transmitted signal to 
determine range and azimuth angle in each of the sweeps 
30 and 32, as further described in step 104. Targets or 
entities 34 in each of the sweeps 30 and 32 are therefore 
associated with polar coordinates(r,6) and a return inten- 
sity level. 



[0042] | n step 102, a first level of data fusion occurs by using 

sensor data, such as range, azimuth angle, and return in- 
tensity, to gate and detect objects from the sweeps 30 
and 32. The entities 34 are gated and paired for each de- 
tected object 36. In so doing the entities 34 may be aver- 
aged or combined using other methods known in the art, 
such that a single resulting target 38 is generated for 
each object 36. In one embodiment of the present inven- 
tion the upramp objects are paired with the downramp 
objects such that they have similar return intensity and 
Doppler frequencies at a particular azimuth angle. 

[0043] | n s tep 104, range, range-rate, and direction of travel are 
determined for each resulting target 38 relative to the 
host vehicle 12. Range rate or relative velocity of an object 
may be determined in response to Doppler shift as known 
in the art. 

[0044] Figure 3 illustrates an example of steps 100-104. A first 
object 40 is shown having a first upramp target 42 and a 
first downramp target 44, along with a second object 46 
having a second upramp target 48 and a second down- 
ramp target 50. The upramp targets 42 and 48 and the 
downramp targets 44 and 50 are gated and paired for 
each object 36 to form resulting targets 38, respectively. 



The resulting targets 38 have ranges R and R 2 and az- 
imuth angles 6 i and respectively. 

[0045] Referring again to Figure 2, in step 106, the image- 
generating unit 16 captures and stores image signals. 
Each image signal has an image frame 52, as best seen in 
Figure 4, with a field-of-view 54 and including images or 
image of targets 55, corresponding to the objects 36 that 
were detected in step 100. 

[0046] | n the following Figures including Figure 4, objects, re- 

gions-of-interest (ROIs), blocks, etc. are enlarged and are 
not to scale for description and understanding purposes, 
all of which in actuality may be respectively smaller and 
vary relatively in size. 

[0047] | n s tep 108, centers-of-reflection 56 are determined in 
response to each of the object detection signals. 

[0048] in step 108A, the polar coordinates for each target 38 are 
transformed into a world coordinate system. The coordi- 
nates for each target 38 correspond to the center- 
of-reflection 56 representing centers of objects within an 
electro-magnetic sensor field. For example, the first range 
R i and the first azimuth angle 8 for the first object 40 is 
converted into World coordinates (XI , Yl , and Zl ). 

WW w 

[0049] in step 108B, perspective transformation is performed on 



each of the World coordinates of each target 38, which re- 
sults in transferring the centers-of-reflection 56 from the 
World coordinate system into an image coordinates sys- 
tem. For example, for the first object 40, Cartesian coor- 
dinates XI , Yl , and Zl are converted in to corre- 

w w w 

sponding image coordinates XI., Yl.. The image coordi- 
nates for each target 38 are effectively overlaid on the im- 
age frame 52, as best seen in Figure 5. The image coordi- 
nates XI , Yl represent a center-of-reflection 58 for the 

i i 

first object 40. 

[0050] | n s tep 108C, an initial difference parameter d Q is deter- 
mined for each of the objects 36. The initial value of the 
difference parameter d Q is an approximate number. The 
value of parameter d Q is proportional to the velocity and 
range of an object relative to the host vehicle 12 and is 
also proportional to an average parameter d and is ad- 

avg 

justed for any associated errors. The average difference 
parameter d , which is part of the sensor difference sig- 

avg 

nal and is generated in response to a set of difference pa- 
rameters d , the centers-of-reflection 56, and centers- 

i 

of-intensity 59. The centers-of-intensity 59 are intensity 
averaged center designators for objects in each ROI 60, as 
described below. Average parameter d is equal to zero 

avg 



for a first time period and accounts for errors between the 
centers-of-reflection 56 and the associated centers- 
of-intensity 59 for each object 36. Average parameter d 
is determined using difference vectors obtained from pre- 
vious data, for example using radar frame 1 and image 
frame 1-15. The average parameter d is also modified 
by velocity and size of an object as detected in an image 
frame. An approximate circle or range is formed around 
the centers-of-reflection 56 having a radius d. 
[° 051 ] In step 110, the ROIs 60 are defined around each of the 
centers-of-reflection 56 for each object 36. The size of 
the ROIs 60 are determined in response to apriori data. 
The ROIs 60 may be of varying size and shape. The 
present invention uses the electro-magnetic sensors 14 to 
provide a starting reference point, such as each centers- 
of-reflection 56, in an image plane for each detected ob- 
ject. The ROIs 60 are generated around the centers- 
of-reflection 56. The size of each initial ROI 60 is deter- 
mined in response to apriori data. For example, in one 
embodiment of the present invention height and width of 
the initial ROI 60 corresponds with height and width of a 
large manufactured vehicle. The apriori data may also in- 
clude object velocity and range relative to the host vehicle 



12. Object dimensions may be converted into a number of 
pixels in X and Y directions of an image plane and used in 
conjunction with object range from the host vehicle 12 to 
compute size of the initial ROI 60. In general, the larger 
the relative range of an object from the vehicle 12 the 
smaller the ROI. 

[0052] For a next or subsequent set of image frames, the corre- 
sponding size of the ROIs 60 is adjusted in response to 
speed and direction of travel of the detected objects 36 
with respect to the host vehicle 12, as illustrated in Fig- 
ures 7 A and 7B by predicted ROIs 70 for detected objects 
36. Figure 7A is an example illustration of the objects 36 
moving away from the vehicle 12 and Figure 7B is an ex- 
ample of the objects 36 becoming closer to the vehicle 
12. The size of the ROIs increases when the detected ob- 
ject 36 are becoming closer to the host vehicle 12 and de- 
creases when the detected objects are moving away from 
the host vehicle 12. As the detected objects 36 are be- 
coming closer to the host vehicle 12 their corresponding 
image increases in size and vice versa when the detected 
objects 36 are moving away from the host vehicle 12. The 
rate at which the ROIs 60 increase or decrease also de- 
pends on the speed of the detected objects 36 relative to 



the host vehicle 12. 
[0053] | n s tep 112, a centroid 62, as seen in Figure 5, is deter- 
mined for each object 40 corresponding to the centers- 
of-intensity 55. The controller 18 may use image- 
processing techniques, such as image filtering (low, high 
pass), edge detection, segmentation, or other processing 
known in the art on the image frame 52, to detect an ob- 
ject in the ROIs 60. The centroids 62 are determined in 
response to contour, height, width, area, and pixel inten- 
sity of each object detected in the ROIs 60 or the centroid 
62 may be determined by determining an average 
weighted center of pixel intensity for the object detected 
in ROIs 60. 

[0054] Referring now also to Figure 6, in step 114, a set of vec- 
tors 64, represented by c^-d^, are determined between 
the centers-of-reflection 56 and the corresponding cen- 
troids 62, for image frames 65 and corresponding radar 
frames 66. Each vector 64 has an associated distance 
value or difference parameter d , where i is the vector 

i 

number. The controller 18 determines the differences d 

i 

between the centers-of-reflection 56 and the correspond- 
ing centroids 62 for each image frame 65 as associated 
with a radar frame 66. 



[0055] | n step 116, the controller 18 compares the difference be- 
tween each of the vectors 64 with the average parameter 
d to determine whether they are less than or equal to 

avg 

the average parameter d . 

avg 

[0056] | n step 116A, when the vectors 64 are less than or equal 
to the average parameter d then the object in the cor- 

avg 

responding radar frame is the same as an object detected 
in a corresponding image frame. In so doing, the present 
invention provides improved gating of associated objects 
targets in two dissimilar modalities or dissimilar sensors. 
Average parameter d is also used in determining an ini- 

avg 

tial value difference parameter d Q for the next sequential 
set of radar frames and corresponding image frame. 
[0057] in step 116B, when a difference parameter d. is greater 
than the average parameter d than that difference pa- 

avg 

rameter d is discarded. In one embodiment of the present 

i 

invention, sensor data is aligned and associated from two 
dissimilar sensors having different scan rates. In one em- 
bodiment of the present invention, the electro-magnetic 
sensors 14 have a scan rate of approximately two frames 
per second and the electro-optical sensor 16 has a scan 
rate of approximately 30 frames per second. To align and 
associate data from sensors 14 and 16 image frames 65 



are associated with each radar frame 66. Thus, the set of 
vectors 64 for a first radar frame RF has 15 associated 

A 

image frames IF -IF 

3 Al A15 

[0058] | n step 118, the controller 18 determines difference pa- 
rameters d between the center-of-reflection and the cen- 

i 

troid obtained for the fifteen radar frames relative to the 
first image frame. Once all the difference parameters d. 
have been determined then an average difference param- 
eter value d is determined in response to the difference 

avg 

parameters d and is used as apriori information for com- 

i 

putation of difference parameters d for a subsequent set 

i 

of radar frames. Steps 101-118 are repeated for each de- 
tected object until the detected object is no longer in 
close proximity of the host vehicle 12 i.e. all the differ- 
ence parameters d. are greater than the initial distance d Q . 
In one embodiment of the present invention, an object is 
determined to be in close proximity when at least approx- 
imately 60% of the difference parameters d are less than 

i 

or equal to the initial difference d Q . As the percentage of 
difference parameters d that are less than or equal to the 

i 

initial difference d Q increases the probability that the ob- 
ject is of concern increases and vice versa. In another em- 
bodiment of the present invention, when less than 60% of 



the difference parameters d are less than or equal to the 

i 

initial difference d Q then the detected object is classified 
as a "ghost object"or a false object. When a ghost object 
or a false object is detected an error signal is generated 
and the controller 18 returns to step 100. 
[0059] For example, when the current set of image frames is IF 
, having corresponding radar frame RF , the next or 

X "™ X 5 

subsequent set of image frames is IF and has corre- 

3 Bl-15 

sponding radar frame RF g . The subsequent set of image 
frames IF repeat the same procedure as performed 
for the current set of image frames IF . Polar coordi- 

3 Al-15 

nates for the subsequent radar frame RF of the detected 

M B 

object are determined followed by performing coordinate 
system transform and perspective transform to obtain 
new or subsequent image coordinates IF , as stated in 

M 3 Bl-15 

step 108. 

[0060] | n s tep 120, a subsequent set of ROIs are defined for each 
detected object. The new ROIs change between sequential 
image frames. The subsequent set of ROIs rather than be- 
ing determined upon completion of step 118 may be de- 
termined in step 110. The subsequent set of ROIs may be 
determined, as described in step 110, by determining 
centers-of-intensity for each image frame with respect to 



a subsequent radar frame. When determining the subse- 
quent set of ROIs, the average parameter d is deter- 

avg 

mined from a previous set of ROIs to increase accuracy in 
determining whether a detected object is a potential 
threat to the host vehicle 12. The radar frame designator j 
is set equal to the current value of j plus one. When an er- 
ror is detected the controller 18 may consider an object to 
be a false or ghost object and sets the radar frame desig- 
nator j equal to one or performs some other task as 
known in the art. 
[0061] | n s tep 122, the detected real objects 36 may be classified 
and tracked in response to the sensor difference signal. 
The controller 18 may provide collision warning and 
countermeasure performance in response to object infor- 
mation including the object classification. Depending 
upon vehicle and object relative positions, velocities, and 
accelerations, the controller 18 may determine whether to 
indicate to a vehicle operator, via the indicator 26, of a 
potential collision, or may determine to perform a passive 
countermeasure or an active countermeasure. Upon com- 
pletion of step 124 the controller 18 returns to step 101 
when the value of the radar frame designator is greater 
than one to continue tracking the objects 36, otherwise 



the controller returns to step 100. 
[0062] The above-described steps are meant to be an illustrative 
example; the steps may be performed sequentially, syn- 
chronously, or in a different order depending upon the 
application. 

[0063] The present invention provides a collision warning and 

countermeasure system that utilizes object detection sen- 
sors in conjunction with image generating sensors to 
minimize and simplify image processing and more effi- 
ciently classify and track objects. The multiple sensor data 
fusion architecture of the present invention reduces 
amount of image processing by processing only selected 
areas of an image frame as determined in response to in- 
formation from electromagnetic sensors. 

[0064] while the invention has been described in connection with 
one or more embodiments, it is to be understood that the 
specific mechanisms and techniques which have been de- 
scribed are merely illustrative of the principles of the in- 
vention, numerous modifications may be made to the 
methods and apparatus described without departing from 
the spirit and scope of the invention as defined by the ap- 
pended claims. 



